Method and Apparatus For Correlating Simulation Models With Physical Devices Based on Correlation Metrics

ABSTRACT

A method, program product, and apparatus are provided to correlate operation of a first system and a second system. A first set of metrics are generated for a first set of data produced during operation of the first system. A second set of metrics are generated that correspond to each of metrics in the first set of metrics from a second set of data produced during operation of the second system. A correlation score is computed for the first and second systems based on differences between the first set of metrics and the second set of metrics. The first set of metrics, the second set of metrics, and the correlation score are presented on the display to indicate a similarity of operation of the first and second systems.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

The U.S. Government may have a paid-up license in this invention and theright in limited circumstances to require the patent owner to licenseothers on reasonable terms as provided by the terms of Contract No.NO0178-04-D-4123-AE&S-EK01 awarded by The Department of the Navy(Strategic Systems Programs).

BACKGROUND

1. Technical Field

The present invention embodiments pertain to data correlation. Inparticular, the present invention embodiments pertain to correlating asimulation model with a physical device based on correlation metrics andplotting the metrics to indicate variance between the simulation modeland device.

2. Discussion of Related Art

Numerical analysis is widely used to find approximate solutions forproblems lacking exact solutions. In mathematics, these problems canrange from approximating the values of roots, e.g., square roots, andthe value of π, to evaluating integrals and differential equations. Themathematical problems may represent many real-world problems for whichnumerical analysis can provide solutions (e.g., weather prediction,scheduling and pricing algorithms, crash simulation, explosion modeling,etc.).

Numerical analysis has been used to approximate solutions by usinginterpolation and extrapolation. Many techniques have been developed toprovide approximations to various problems of interest. Initially,various functions and their approximations at certain points, computedto various degrees of precision, were available in reference books. Withthe advent of modern computers, many problems can now be modeled. Forexample, a finite element method is incorporated into software forfinite element analysis (FEA) and can provide approximate solutions forcomplex heat flow problems, magnetic flux calculations, or to modelstructural failures. Other software applications may solve systems witha large number of equations or Markov chains.

For real-world applications, simulation is highly desirable because ofthe lower cost or the impracticality of building a prototype. In oneexample, a simulation is preferred over building a physical device ortest case for each test scenario. In other examples, building a testcase is impractical due to scaling effects (e.g., it is impractical tobuild a physical prototype to simulate world weather patterns or tosimulate earthquakes).

While simulation of real-world systems can provide more practical andless expensive solutions when compared to actual physical prototypes,the simulation is almost always flawed because assumptions andapproximations are introduced into the simulation mathematical model tosimplify the processing requirements. In order to compare the simulationwith the real-world system, a limited quantity of actual data may beplotted against simulation data to enable visual interpretation of thecorrelation.

SUMMARY

According to a present invention embodiment, a method, program product,and apparatus are provided to correlate operation of a first system anda second system. Briefly, correlation metrics are generated for a firstset of data and a second set of data, a correlation score is computedbetween the correlation metrics, and a plot of the correlation metricsand correlation score is displayed. The apparatus includes at least anoutput module and a processor. The output module is coupled to a displayto present the plurality of correlation metrics and the correlationscore. The processor controls the presentation of the plurality ofcorrelation metrics and the correlation score. The processor includes acorrelation module to generate a first set of metrics from the first setof data produced during operation of the first system, generate a secondset of metrics that correspond to each of the first set of metrics fromthe second set of data produced during operation of the second system,and compute a correlation score for the first and second systems basedon differences between the first set of metrics and the second set ofmetrics. The first set of metrics, the second set of metrics, and thecorrelation score are presented on the display via the output module toindicate the similarity of operation of the first and second systems.

The above and still further features and advantages of the presentinvention will become apparent upon consideration of the followingdetailed description of example embodiments thereof, particularly whentaken in conjunction with the accompanying drawings wherein likereference numerals in the various figures are utilized to designate likecomponents.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example general purpose computer with acorrelation module configured to generate correlation metrics inaccordance with an embodiment of the present invention.

FIG. 2 is a graphical illustration of an example generic radial plot ofa plurality of correlation metrics in accordance with an embodiment ofthe present invention.

FIG. 3 is a procedural flowchart illustrating the manner in which ageneral purpose computer generates correlation metrics and computes acorrelation score in accordance with an embodiment of the presentinvention.

FIG. 4 is a graphical illustration of a response curve for a low passfilter used in connection with generating correlation metrics inaccordance with an embodiment of the present invention.

FIG. 5 is a graphical illustration of an example plot of microstrainagainst time for test data recorded from a strain gauge and simulationdata generated from a computer model of the strain gauge.

FIG. 6 is a graphical illustration of a radial plot of a plurality ofcorrelation metrics that are generated in accordance with an embodimentof the present invention for the test data and simulation dataillustrated in FIG. 4.

FIG. 7 is a graphical illustration of an example plot of microstrainagainst time for test data recorded from a strain gauge and simulationdata generated from an updated computer model of the strain gauge.

FIG. 8 is a graphical illustration of a radial plot of a plurality ofcorrelation metrics that are generated in accordance with an embodimentof the present invention for the test data and updated simulation dataillustrated in FIG. 7.

FIG. 9 is a graphical illustration of an example plot of microstrainagainst time for test data recorded from a strain gauge and simulationdata generated from a further updated computer model of the straingauge.

FIG. 10 is a graphical illustration of a radial plot of a plurality ofcorrelation metrics that are generated in accordance with an embodimentof the present invention for the test data and the further updatedsimulation data illustrated in FIG. 9.

FIG. 11 is a graphical illustration of a radial plot of a plurality ofcorrelation metrics that are generated in accordance with an embodimentof the present invention for test data and simulation data derived foran accelerometer.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

In order to determine how well a simulation models a real-world system,model developers have devised various tests or metrics to compare howwell a simulation models a real-world system, e.g., simulation data maybe compared to actual test data using test statistics or metrics. By wayof example, a structure may be fitted with strain gauges and then besubjected to an impulse or force, e.g., a mass is dropped on thestructure. The outputs of the test strain gauges are recorded. Thestructure and strain gauge outputs may be simulated in software usingFEA. The simulated strain gauge outputs are then compared to therecorded test strain gauge outputs using a metric, e.g., a root-meansquare (RMS) error between the simulation strain gauge outputs and teststrain gauge outputs. The lower the RMS error, the better the simulationis said to “fit” the real-world test.

Current methods to compare simulation data with test data may use only asingle metric and do not always provide a meaningful way to interpretthe simulation results relative to the test results. The continuallyincreasing complexity of computer models leads to increasingly complexanalytical predictions that are difficult to correlate with physicaltest data. As the number of metrics used for comparison increases, thedecision process as to whether or not the simulation fits thereal-world, becomes more complex.

The present invention embodiments provide a methodology and apparatusthat provides a consistent and objective approach to quantitativelycorrelate test data, simulation results, or test data and simulationresults. Also provided is a concise way to display correlation metricsand compute an associated correlation score based on the correlationmetrics.

An example device configured to generate and display correlation metricsis illustrated in FIG. 1. Specifically, the device comprises a generalpurpose computer 100 with a processor 110, a network interface 120, aplurality of input/output (IO) modules 130, a display 140, and acorrelation module 300. General purpose computer 100 may further includeother application specific components (e.g., a sensor interface moduleor other data gathering devices for the collection of test data).Processor 110 is preferably implemented by a conventional microprocessoror controller and controls the various components and modules inaccordance with the techniques described below. Network interface 120 ispreferably implemented by a conventional network interface, e.g., anetwork interface card (NIC), and provides network communications forgeneral purpose computer 100. IO modules 130 are preferably implementedby conventional IO components typically included on a motherboard ofgeneral purpose computer 100 and may include IO interfaces for akeyboard, a computer mouse, and a video output for a computer displaymonitor e.g., for output to display 140.

Generally, correlation module 300 contains the intelligence to generatecorrelation metrics associated with two or more sets of data and displaythe metrics to a user according to a present invention embodiment.Module 300 may be in the form of computer instructions that are executedby processor 110 or may be implemented in hardware, e.g., logic builtinto an application specific integrated circuit (ASIC) or logicprogrammed into a field programmable gate array (FPGA). Once correlationmodule 300 generates the various correlation metrics, then correlationmodule 300 computes a correlation score that indicates the relativecloseness or fit between the two or more sets of data. Correlationmodule 300, by way of processor 110, causes the correlation metrics andcorrelation score to be presented to the user on display 140 in an easyto understand form.

An example radial or spider plot 200 of the correlation metrics andscore is depicted in FIG. 2. A series of metrics 1-8 are being comparedfor two sets of data. Spider plot 200 has a series of concentric dashedrings or circles 210, and a solid baseline ring 220. The rings 210 and220 are labeled with a percentage deviation from baseline 230. Thebaseline ring 220 represents the first set of metrics, e.g., metricsgenerated from test data collected for a real-world system. A second setof metrics 240 are plotted as a thick line relative to baseline ring220. The second set of metrics 240 may be generated from data collectedfrom a simulation of the real-world system. Alternatively, the two setsof data could come from two simulation runs and the simulation metricsare compared to each other or the two sets of data could come from tworeal-world tests and the test metrics are compared to each other.

General purpose computer 100 preferably employs Graphical User Interface(GUI) functionality to display spider plot 200 with normalized metricsplotted against the baseline. Spider plot 200 depicts the first set ofmetrics 220 as a baseline of zero percent and the second set of metrics240 is plotted as a percentage deviation from the baseline metrics 220.Spider plot 200 shows deviations from the baseline ring 220 for thesecond set of metrics 240 of approximately +5%, 0%, +8%, −1%, 0%, −7%,+10%, and −3% for metrics 1-8, respectively. Metrics with little or nodeviation from the baseline, e.g., metrics 2, 4, and 5, correlateclosely with the real-world system being modeled. Metrics with a largerdeviation, e.g., metrics 3, 6, and 7, do not correlate as closely withthe real-world system and indicate areas in which the simulation modelmay be improved.

If all metrics 1-8 are deemed to be of equal importance, then theabsolute values of all the deviations from the baseline could be summedto attain an overall correlation score. Summing the absolute values ofthe above deviations from the baseline yields a correlation score of 34.If the metrics are not equally important, then weights could be assignedto each metric before computing the correlation score, i.e., a weightedsummation or average may be used to compute the correlation score. Thecorrelation score may also be displayed on the display 140.

As can be seen from FIG. 2, a relatively large number of correlationmetrics for different sets of data can be displayed relative to oneanother. Additional axes or spider plots may be used to compareadditional metrics. Spider plot 200 is drawn using concentric rings 210and 220, but could be drawn using concentric polygons, e.g., for eightmetrics concentric octagons could be used.

A procedural flowchart generally outlining the manner in whichcorrelation module 300 generates correlation metrics and computes acorrelation score according to a present invention embodiment isillustrated in FIG. 3. For purposes of describing the flowchart, it ispresumed that a first and a second set of data have already beengenerated or captured, e.g., a set of simulation data and a set of testdata. The first and second sets of data may be preprocessed at step 310.In particular, prior to producing the metrics, the test data andsimulation data may be subject to preprocessing by an application,whereby the amount of data may be reduced. Prior to evaluating theperformance metrics, the application formats the data to ensureconsistent calculation of all metrics. The application may provide auser with the ability to convert data between various units using, e.g.,a dropdown box. In one example, strain data may be converted tomicro-strain based upon the user selection in the dropdown box.

The application may also provide for time alignment of the data sets toensure a reliable correlation score. The minimum time of all data setsare set to 0.0 seconds. This eliminates test data preceding 0.0 secondsor data that are not pertinent to the test. The application may also setthe maximum time for each data set by using the lesser of the twomaximum time values from each data file. This will allow the analysis ofthe maximum amount of data for the calculations.

Data for both the simulation and test may be filtered. For the straingauge model, it is useful to filter the data at 20 kHz using aButterworth low-pass filter (LPF). Design of the filter may include 3 dBof ripple in the pass-band and 10 decibels (dB) of attenuation in the 5kHz stop-band. Filter order may be calculated using, for example, theMATLAB function buttord. With the order of the filter defined, togetherwith the pass-band and stop-band definition, the MATLAB function,butter, calculates the filter coefficients. These filter coefficientsdefine the filter and are used by the MATLAB filter command, filtfilt,to filter the data forwards and backwards, mitigating any phasedistortion. An example response curve for such a low-pass filter isshown in FIG. 4.

In another example, the application may perform data decimation as thefinal step in data reduction prior to calculating the metrics. Data maybe decimated to 100 kHz, thereby generating congruent sampling rates forthe simulation and test data. Identical sampling rates and time durationensures that any Fast Fourier Transforms (FFTs) of both test andsimulation data have equal number and length frequency bands.

Before metrics can be generated for time series data, the start times ofeach set of data must be aligned to each other. The start times for eachset of data may be normalized to a predetermined value, e.g., zero, orthe data may be time aligned, one relative to another. Sample rates andperiods should also be accounted or otherwise compensated for whengenerating metrics for correlation with each other. A first set ofmetrics is generated from a first set of data, e.g., test data, at step320. A second set of metrics is generated from a second set of data,e.g., simulation data, at step 330. Since the metrics are to becompared, the types of metrics generated from the first and second setsof data are the same.

Two example sets of data for a strain type application that may be usedto generate correlation metrics are shown in FIG. 5. Test data 510 andsimulation data 520 are plotted as microstrain versus time. Test data510 was captured from a strain gauge labeled s247 that is coupled to astructure. A force was applied to the structure and the response fromstrain gauge s247 was recorded. Simulation data 520 were derived from amathematical model of the structure and strain gauge s247. Simulationdata 520 may be generated using commercially available mathematicssimulation software, such as MATLAB or Mathematica that runs on ageneral purpose computer.

Correlation metrics were generated from both sets of data 510 and 520. Aspider plot of the correlation metrics generated in year 1 is shown inFIG. 6. The correlation metrics for the test data 510 are plotted as adodecagon or baseline ring 610. Simulation correlation metrics 620 areplotted relative to the baseline 610. The spider plot provides aconvenient way of highlighting highly similar or dissimilar correlationmetrics. Example correlation metrics and corresponding weights aredescribed. By way of example, the metrics include peak to peak, timing,and frequency correlation metrics. However, any suitable metrics andweights may be used.

The Peak (+) Metric is the percent error between the maximum responsevalue of the FEA and the maximum response value of the test data dividedby the maximum response value of the test data. For purposes ofcalculating the correlation score, the Peak (+) Metric is given a weightof 17.5% as shown in Eq. 1.

$\begin{matrix}{{{Peak}\mspace{14mu} ( + )} = {( {17.5\%} ) \times \lbrack {( \frac{{FEA}_{\max} - {Test}_{\max}}{{Test}_{\max}} ) \times 100} \rbrack}} & ( {{Eq}.\mspace{14mu} 1} )\end{matrix}$

The Peak (+) Time Metric is the percent error between an occurrence timeof the maximum response value of the FEA and an occurrence time of themaximum response value of the test data, divided by one millisecond. Theone millisecond denominator normalizes the calculation by mitigating theeffect of small time denominator at an early time or a large timedenominator at a late time that would otherwise be used in thecalculation. The Peak (+) Time Metric is given a weight of 7.5% as shownin Eq. 2.

$\begin{matrix}{{{Peak}\mspace{14mu} ( + )\mspace{14mu} {Time}} = {( {7.5\%} ) \times \lbrack {( \frac{{FEA}_{\max \mspace{11mu} {time}} - {Test}_{\max \mspace{11mu} {time}}}{0.001\mspace{14mu} \sec} ) \times 100} \rbrack}} & ( {{Eq}.\mspace{14mu} 2} )\end{matrix}$

The Peak (−) Metric is the percent error between the minimum responsevalue of the FEA and the minimum response value of the test data dividedby the minimum response value of the test data. For purposes ofcalculating the correlation score, the Peak (−) Metric is given a weightof 17.5% as shown in Eq. 3.

$\begin{matrix}{{{Peak}\mspace{14mu} ( - )} = {( {17.5\%} ) \times \lbrack {( \frac{{FEA}_{\min} - {Test}_{\min}}{{Test}_{\min}} ) \times 100} \rbrack}} & ( {{Eq}.\mspace{14mu} 3} )\end{matrix}$

The Peak (−) Time Metric is the percent error between an occurrence timeof the minimum response value of the FEA and an occurrence time of theminimum response value of the test data, divided by one millisecond. Theone millisecond denominator normalizes the calculation by mitigating theeffect of small time denominator at an early time or a large timedenominator at a late time that would otherwise be used in thecalculation. The Peak (−) Time Metric is given a weight of 7.5% as shownin Eq. 4.

$\begin{matrix}{{{Peak}\mspace{14mu} ( - )\mspace{20mu} {Time}} = {( {7.5\%} ) \times \lbrack {( \frac{{FEA}_{\min \mspace{11mu} {time}} - {Test}_{\min \mspace{11mu} {time}}}{0.001\mspace{14mu} \sec} ) \times 100} \rbrack}} & ( {{Eq}.\mspace{14mu} 4} )\end{matrix}$

The 3 Peaks (+) Metric is the average absolute difference between thesubsequent 3 largest positive response values following the maximumvalue, divided by the average of the test's subsequent 3 largestpositive response values. The 3 Peaks (+) Metric is given a weight of7.5% as shown in Eq. 5.

$\begin{matrix}{{3\mspace{14mu} {Peak}\mspace{14mu} ( + )} = {( {7.5\%} ) \times \lbrack {( \frac{{average}( {{{FEA}_{\max \mspace{14mu} 3} - {Test}_{\max \mspace{11mu} 3}}} )}{{average}( {{Test}_{\max \mspace{14mu} 3}} )} ) \times 100} \rbrack}} & ( {{Eq}.\mspace{14mu} 5} )\end{matrix}$

The 3 Peaks (+) Time Metric is the average absolute difference betweenthe occurrence times of the subsequent 3 largest positive responsevalues following the maximum value, divided by one millisecond. The onemillisecond denominator normalizes the calculation by mitigating theeffect of small time denominator at an early time or a large timedenominator at a late time that would otherwise be used in thecalculation. The 3 Peaks (−) Time Metric is given a weight of 2.5% asshown in Eq. 6.

$\begin{matrix}{{3\mspace{14mu} {Peak}\mspace{14mu} ( + )\mspace{20mu} {Time}} = {( {2.5\%} ) \times \lbrack {( \frac{{average}( {{{FEA}_{\max \mspace{11mu} 3} - {Test}_{\max \mspace{11mu} 3}}} )}{0.001\mspace{14mu} \sec} ) \times 100} \rbrack}} & ( {{Eq}.\mspace{14mu} 6} )\end{matrix}$

The 3 Peaks (−) Metric is the average absolute difference between thesubsequent 3 most negative response values following the minimum value,divided by the average of the test's subsequent 3 most negative responsevalues. The 3 Peaks (−) Metric is given a weight of 7.5% as shown in Eq.7.

$\begin{matrix}{{3\mspace{14mu} {Peak}\mspace{14mu} ( - )} = {( {7.5\%} ) \times \lbrack {( \frac{{average}( {{{FEA}_{\min \mspace{11mu} 3} - {Test}_{\min \mspace{11mu} 3}}} )}{{average}( {{Test}_{\min \mspace{11mu} 3}} )} ) \times 100} \rbrack}} & ( {{Eq}.\mspace{14mu} 7} )\end{matrix}$

The 3 Peaks (−) Time Metric is the average absolute difference betweenthe occurrence times of the subsequent 3 most negative response valuesfollowing the minimum value, divided by one millisecond. The onemillisecond denominator normalizes the calculation by mitigating theeffect of small time denominator at an early time or a large timedenominator at a late time that would otherwise be used in thecalculation. The 3 Peaks (−) Time Metric is given a weight of 2.5% asshown in Eq. 8.

$\begin{matrix}{{3\mspace{14mu} {Peak}\mspace{14mu} ( - )\mspace{14mu} {Time}} = {( {2.5\%} ) \times \lbrack {( \frac{{average}( {{{FEA}_{\min \mspace{11mu} 3} - {Test}_{\min \mspace{11mu} 3}}} )}{0.001\mspace{14mu} \sec} ) \times 100} \rbrack}} & ( {{Eq}.\mspace{14mu} 8} )\end{matrix}$

The Frequency (0-5 kHz) Metric is the percent error between the dominantfrequency components of the FEA output and test output over the 0 to 5kHz band, and divided by the 5 kHz bandwidth. The 5 kHz denominatornormalizes the calculation by mitigating the affect of a smalldenominator at lower frequencies or a large denominator at higherfrequencies that would otherwise be used in the calculation. TheFrequency (0-5 kHz) Metric is given a weight of 10.0% as shown in Eq. 9.

$\begin{matrix}{{{Frequency}\mspace{14mu} ( {0\text{-}5\mspace{14mu} {kHz}} )} = {( {10.0\%} ) \times \lbrack {( \frac{{FEA}_{({0\text{-}5\mspace{14mu} {KHz}})} - {Test}_{({0\text{-}5\mspace{14mu} {KHz}})}}{5\mspace{14mu} {KHz}} ) \times 100} \rbrack}} & ( {{Eq}.\mspace{14mu} 9} )\end{matrix}$

The Frequency (0-5 kHz) Absolute Magnitude Metric is the percent errorbetween the magnitude of the dominant frequency of the FEA and themagnitude of the dominant frequency of the test data over the 0 to 5 kHzband. The Frequency (0-5 kHz) Absolute Magnitude Metric is given aweight of 5.0% as shown in Eq. 10.

$\begin{matrix}{{{Freq}\mspace{14mu} ( {0\text{-}5\mspace{14mu} {kHz}} )\mspace{14mu} {{Abs}.\mspace{14mu} {Mag}.}} = {( {5.0\%} ) \times \lbrack {( \frac{{FEA}_{{mag}{({0\text{-}5\mspace{14mu} {KHz}})}} - {Test}_{{mag}{({0\text{-}5\mspace{14mu} {KHz}})}}}{{Test}_{{mag}{({0\text{-}5\mspace{14mu} {KHz}})}}} ) \times 100} \rbrack}} & ( {{Eq}.\mspace{14mu} 10} )\end{matrix}$

The Frequency (5-10 kHz) Metric is the percent error between thedominant frequency components of the FEA output and test output over the5 to 10 kHz band, and divided by the 5 kHz bandwidth. The 5 kHzdenominator normalizes the calculation by mitigating the effect of asmall denominator at lower frequencies or a large denominator at higherfrequencies that would otherwise be used in the calculation. TheFrequency (5-10 kHz) Metric is given a weight of 10.0% as shown in Eq.11.

$\begin{matrix}{\; {{{Frequency}\mspace{14mu} ( {5\text{-}10\mspace{14mu} {kHz}} )} = {( {10.0\%} ) \times \lbrack {( \frac{{FEA}_{({5\text{-}10\mspace{14mu} {KHz}})} - {Test}_{({5\text{-}10\mspace{14mu} {KHz}})}}{5\mspace{14mu} {KHz}} ) \times 100} \rbrack}}} & ( {{Eq}.\mspace{14mu} 11} )\end{matrix}$

The Frequency (5-10 kHz) Absolute Magnitude Metric is the percent errorbetween the magnitude of the dominant frequency of the FEA and themagnitude of the dominant frequency of the test data over the 5 to 10kHz band. The Frequency (5-10 kHz) Absolute Magnitude Metric is given aweight of 5.0% as shown in Eq. 12.

$\begin{matrix}{{{Freq}\mspace{14mu} ( {5\text{-}10\mspace{14mu} {kHz}} )\mspace{14mu} {{Abs}.\mspace{14mu} {Mag}.}} = {( {5.0\%} ) \times \lbrack {( \frac{{FEA}_{{mag}{({5\text{-}10\mspace{14mu} {KHz}})}} - {Test}_{{mag}{({5\text{-}10\mspace{14mu} {KHz}})}}}{{Test}_{{mag}{({5\text{-}10\mspace{14mu} {KHz}})}}} ) \times 100} \rbrack}} & ( {{Eq}.\mspace{14mu} 12} )\end{matrix}$

Referring back to FIG. 3, the sets of data are normalized relative toeach other at step 340. The metrics for one set of data are normalizedrelative to the other such that one set of metrics forms a baseline. Inone example, each of the metrics in the second set of metrics aremathematically divided by corresponding metrics in the first set ofmetrics, whereby each metric in the first set of metrics become abaseline, e.g., a baseline of zero or one, and each metric in the secondset of metrics is normalized thereto. Metrics generated in equations1-12 are normalized to the test data by dividing the metric bycorresponding test data or a neutral parameter, e.g., 0.001 seconds or 5KHz. Other mathematical operations may be performed on the data sets,e.g., quantization or a Euclidian norm operation.

The normalized metrics are presented, displayed, or otherwise plottedagainst the baseline at step 350, for example, the second set of metrics620 as shown in FIG. 6. A correlation score is computed based ondifferences between the first and second sets of metrics at step 360.The Correlation Score provides the final score, rating the correlationof simulation data with test data. Summing the absolute values of thetwelve weighted metric scores described above produces the CorrelationScore as shown in Eq. 13. Perfect correlation would result in aCorrelation Score of 0.0, while an imperfect correlation score rangesbetween 0.0 and infinity (an upper bound does not exist). Thecorrelation score for the set of model correlation metrics depicted inFIG. 6 is 60.1001.

$\begin{matrix}{{{Correlation}\mspace{14mu} {Score}} = {\sum\limits_{n = 1}^{12}\; {{WeightedMetric}_{n}}}} & ( {{Eq}.\mspace{14mu} 13} )\end{matrix}$

The metrics used for plotting and computing the correlation score arechosen based upon the real-world system that is being modeled. Forexample, weighted performance metrics may be chosen and vetted through apeer review process. The metrics may be chosen to correlate systemresponse in both the time and/or frequency domains. The metrics may beweighted based upon their peer reviewed importance or other industrystandards. The summation of weighted metrics leads to a correlationscore that is a quantitative measure of how well the computational modelrepresents the response of the system based upon the chosen weightingscheme. In addition to correlating current simulation results with testdata, the correlation score provides a baseline to compare futureresults and determine if updates to the computational model have apositive effect.

Improvements to the simulation may be made over time, where the spiderplots indicate the areas of the simulation digressing from the testdata, while the correlations score indicates the “fit” of the simulationmodel. Two sets of data that may be used to generate correlation metricsare shown in FIG. 7. Test data 510 is the same test data that was shownin FIG. 5. The simulation data 710 were generated in year 2 from anupdated simulation model (the mathematical model of the structure andstrain gauge s247). Test data 510 and simulation data 710 are plotted asmicrostrain versus time.

Correlation metrics were generated from both sets of data 510 and 710. Aspider plot of the correlation metrics generated in year 2 is shown inFIG. 8. The correlation metrics for the test data 510 are plotted as thebaseline ring 610 as in FIG. 6. Simulation correlation metrics 810 areplotted relative to the baseline 610. The types of correlation metricsshown in FIG. 8 are the same as those described in connection with FIG.6. The correlation score for the correlation metrics generated in year 2is 20.4154. This correlation score is lower than the correlation scoregenerated in year 1 and indicates that simulation model has improvedsince year 1.

Still further improvements to the simulation may be made based on thespider plot. Two sets of data that may be used to generate correlationmetrics are shown in FIG. 9. Test data 510 is the same test data thatwas shown in FIGS. 5 and 7. The simulation data 910 were generated inyear 3 from a further updated simulation model. Test data 510 andsimulation data 910 are plotted as microstrain versus time.

Correlation metrics were generated from both sets of data 510 and 910. Aspider plot of the correlation metrics generated in year 3 is shown inFIG. 10. The correlation metrics for the test data 510 are plotted asthe baseline ring 610 as in FIGS. 6 and 8. Simulation correlationmetrics 1010 are plotted relative to the baseline 610. The types ofcorrelation metrics shown in FIG. 10 are the same as those described inconnection with FIG. 6. The correlation score for the correlationmetrics generated in year 3 is 15.1404. This correlation score is lowerthan the correlation score generated in year 2 and indicates thatsimulation model has improved since year 2.

In another example, accelerometers may be attached to the structure inaddition to or in lieu of the strain gauges. After a force is applied tothe structure, the accelerations from the accelerometer are recorded.Simulation data are derived from a mathematical model of the structureand the accelerometer. Correlation metrics were generated from therecorded accelerometer data and the simulation data. A spider plot ofthe correlation metrics is shown in FIG. 11. The correlation metrics forthe acceleration test data are plotted as the baseline ring 1110.Simulation correlation metrics 1120 are plotted relative to the baselinering 1110. Starting with the Peak (+) Metric and preceding clockwise,each of the correlation metrics shown in FIG. 11 will now be described.

The Peak (+/−) Metrics, Peak (+/−) Time Metrics, Frequency (0-5/5-10kHz), and Frequency (0-5/5-10 kHz) Absolute Magnitude Metrics operateessentially as described above. In this example, the 3 Peak Metrics arenot used and two new metrics have been added that are more germane tomodeling accelerometer simulations. The new metrics are RMS errorbetween the test data and simulation data and the Los Alamos NationalLaboratory (LANL) Shock Response Spectrum (SRS).

The RMS metric is the percent error between the RMS value of the FEAresults and the RMS value of the test data divided by the RMS value ofthe test data. For purposes of calculating the correlation score, theRMS Error metric is given a weight of 17.5% as shown in Eq. 14. TheRMS(FEA) used in Eq. 14 is calculated according to Eq. 15.

$\begin{matrix}{{{RMS}\mspace{14mu} {Error}} = {( {17.5\%} ) \times \lbrack \frac{{{RMS}({FEA})} - {{RMS}({Test})}}{{RMS}({Test})} \rbrack}} & ( {{Eq}.\mspace{14mu} 14} ) \\{{{RMS}({FEA})} = \sqrt{\frac{{FEA}_{1}^{2} + {FEA}_{2}^{2} + \ldots + {FEA}_{n}^{2}}{n}}} & ( {{Eq}.\mspace{14mu} 15} )\end{matrix}$

The LANL SRS metric is the percent error between the FEA results and thetest data using the SRS_(Factor), where the SRS_(Factor) is the averagedifference between computer simulation results and test data for eachpeak response comprising the SRS. The use of common logarithms in theSRS_(Factor) calculation ensures positive integer values. For purposesof calculating the correlation score, the LANL SRS Metric is given aweight of 17.5% as shown in Eq. 16. The SRS is a plot of the maximumpeak response corresponding to a specific forcing function to a singledegree-of-freedom (SDOF) system as a function of the natural frequencyof the system. Numerical analysis is used to compute the peak responseof the SDOF for the specific forcing function. In this example, theforcing function input is acceleration data recorded during testing orpredicted by the computer model. Eqs. 17 and 18 provide an examplecalculation of the SRS_(Factor) used in Eq. 16.

LANL SRS=(17.5%)×[SRS_(FACTOR)−1]  (Eq. 16)

SRS_(FACTOR)=average(10^(|Δ) ^(SRS) ^(|))  (Eq. 17)

Δ_(SRS)=log₁₀FEA_(SRS)=log₁₀Test_(SRS)  (Eq. 18)

It will be appreciated that the embodiments described above andillustrated in the drawings represent only a few of the many ways ofimplementing a method and apparatus for correlating simulation modelswith physical devices based on correlation metrics.

The functions of the general purpose computer employed by the presentinvention embodiments may be implemented and/or performed by anyquantity of any personal or other type of computer system (e.g.,IBM-compatible, Apple, Macintosh, laptop, rack or cluster systems, palmpilot, etc.) or processing device (e.g., microprocessor, controller,circuitry, etc.), and may include any commercially available operatingsystem (e.g., Windows, OS/2, Unix, Linux, etc.) and any commerciallyavailable or custom software (e.g., browser software, communicationssoftware, server software). These systems may include any types ofmonitors and input devices (e.g., keyboard, mouse, touch screens, etc.)to enter and/or view information.

Databases for storing test and simulation data may be implemented by anyquantity of any type of conventional or other databases (e.g.,relational, hierarchical, etc.) or storage structures (e.g., files, datastructures, disk or other storage, etc.). The databases may store anydesired information arranged in any fashion (e.g., tables, relations,hierarchy, etc.). The databases may be queried with any quantity ortype, and request any type of database or other operation. The queriesmay include any desired format and may be directed toward anyconventional or other database or storage unit.

The correlation metrics may be of any type or quantity. The correlationmetrics may be grouped or combined in any manner and generated anynumber of times, e.g., using Monte Carlo simulation. The correlationscore may be computed in any manner suitable for indicating a similarityof operation between two or more systems. The systems may be of any typeor quantity, and represent real-world systems, simulations of real-worldsystems, or both. Mathematical operations performed on the underlyingdata, the correlation metrics, and the correlation score may be of anytype including matrix operations, quantization, normalization, etc., toproduce an indication of similarity of operation between two or moresystems. The display of the correlation metrics and score may be radialplots, spider plots, nested plots (with drill-down options via a GUI),or any visual representation for presenting a plurality of metrics andthe associated correlation score.

It is to be understood that the software (e.g., the correlation moduleor logic) for the computer systems and/or processing devices of thepresent invention embodiments (e.g., the general purpose computers) maybe implemented in any desired computer language (e.g., ADA, C, C++, C#,Java, etc.) and could be developed by one of ordinary skill in thecomputer arts based on the functional descriptions contained in thespecification and flow charts illustrated in the drawings. The softwareis extensible and may be an add-on module to various simulationpackages, e.g., Ansys FLUENT®, Abaqus®, Simulation Environment andResponse Program Execution Nesting Tool (SERPENT), Aeroheating andThermal Analysis Code (ATAC), etc. Further, any references herein ofsoftware performing various functions generally refer to computersystems or processors performing those functions under software control.

Furthermore, the software of the present invention embodiments may beavailable on a recordable medium (e.g., magnetic or optical mediums,magneto-optic mediums, floppy diskettes, CD-ROM, DVD, memory devices,etc.) for use on stand-alone systems or systems connected by a networkor other communications medium, and/or may be downloaded (e.g., in theform of carrier waves, packets, etc.) to systems via a network or othercommunications medium.

The computer systems and/or processing devices of the present inventionembodiments may alternatively be implemented by any type of hardwareand/or other processing circuitry. The functions of the computer systemsand/or processing devices may be implemented by logic encoded in one ormore tangible media (e.g., embedded logic such as an applicationspecific integrated circuit (ASIC), digital signal processor (DSP)instructions, software that is executed by a processor, etc.). Thus,functions of the computer systems and/or processing devices may beimplemented with fixed logic or programmable logic (e.g.,software/computer instructions executed by a processor) by way of aprogrammable processor, programmable digital logic (e.g., fieldprogrammable gate array (FPGA)) or an ASIC that comprises fixed digitallogic, or a combination thereof.

The various functions of the computer systems and/or processing devicesof the present invention embodiments may be distributed in any manneramong any quantity of software and hardware modules or units, processingor computer systems and/or circuitry, where the computer or processingsystems may be disposed locally or remotely of each other andcommunicate via any suitable communications medium (e.g., LAN, WAN,Intranet, Internet, hardwire, modem connection, wireless, etc.). Forexample, the functions of the present invention embodiments may bedistributed in any manner among several computer systems (e.g., testdata may be processed on one computer, simulation data on a secondcomputer, while correlation metrics may be generated on a thirdcomputer). The software and/or algorithms described above andillustrated in the flow charts may be modified in any manner thataccomplishes the functions described herein. In addition, the functionsin the flow charts or description may be performed in any order thataccomplishes a desired operation.

The computer systems of the present invention embodiments may includeany conventional or other communications devices to communicate over thenetworks via any conventional or other protocols. The computer mayutilize any type of connection (e.g., wired, wireless, etc.) for accessto the network. The communication over the communication networks may beimplemented by any quantity of any type of communications protocol(e.g., TCP/IP, SONET, MAC, ATM, IP/Ethernet over WDM, etc.) operablewithin or outside the OSI model.

It is to be understood that the present invention embodiments are notlimited to the applications disclosed herein, but may be utilized forany device or system that is to be modeled. Further, the presentinvention embodiments may be used to generate metrics in real-time andmay be implemented using a real-time operating system. Such systems mayinclude earthquake or weather modeling in which test data areenvironmentally collected in real-time. In one example, weathersimulation may be run concurrently with real-time weather datacollection. Near real-time correlation metrics may be generated for thesimulation. The real-time correlation metrics may be fed back into thesimulation such that an adaptive simulation model may be improved. Thecorrelation metrics and score may be watched in real-time, e.g., byupdating the spider plots and scores at a predetermined rate.

From the foregoing description, it will be appreciated that theinvention makes available a novel method and apparatus for correlatingsimulation models with physical devices or physical systems based oncorrelation metrics.

Having described preferred embodiments of a new and improved method andapparatus for correlating simulation models with physical devices orphysical systems based on correlation metrics, it is believed that othermodifications, variations and changes will be suggested to those skilledin the art in view of the teachings set forth herein. It is therefore tobe understood that all such variations, modifications and changes arebelieved to fall within the scope of the present invention as defined bythe appended claims.

1. An apparatus to correlate operation of first and second systemscomprising: an output module coupled to a display to present a pluralityof correlation metrics and a correlation score for said first and secondsystems; and a processor to control said presentation of said pluralityof correlation metrics and said correlation score, wherein saidprocessor includes a correlation module to: generate a first set ofmetrics for a first set of data produced during operation of said firstsystem; generate a second set of metrics that correspond to each of saidfirst set metrics from a second set of data produced during operation ofsaid second system; compute a correlation score for said first andsecond systems indicating a similarity of operation based on differencesbetween said first set of metrics and said second set of metrics; andpresent said first set of metrics, said second set of metrics, and saidcorrelation score on said display via said output module to indicatesaid similarity of operation.
 2. The apparatus of claim 1, wherein saidcorrelation module is configured to normalize said second set of metricswith respect to said first set of metrics such that said first set ofmetrics forms a baseline.
 3. The apparatus of claim 2, wherein saidnormalized second set of metrics are presented on said display relativeto said baseline.
 4. The apparatus of claim 3, wherein said normalizedsecond set of metrics and said baseline are presented as a radial plotcomprising an axis for each corresponding metric, wherein each metric insaid normalized second set of metrics is presented relative to acorresponding metric in said baseline.
 5. The apparatus of claim 4,wherein said radial plot comprises a scale to indicate the relativedifferences between said second set of metrics and said first set ofmetrics.
 6. The apparatus of claim 2, wherein said correlation score iscomputed based on differences between said normalized second set ofmetrics and said baseline.
 7. The apparatus of claim 2, wherein saidcorrelation score is computed based on weighted differences between saidnormalized second set of metrics and said baseline.
 8. The apparatus ofclaim 1, wherein said first set of metrics and said second set ofmetrics are presented as a radial plot comprising an axis for eachcorresponding metric, wherein each metric in said second set of metricsis presented relative to a corresponding metric in said first set ofmetrics.
 9. The apparatus of claim 1, wherein said metrics within saidfirst and second sets are weighted.
 10. The apparatus of claim 1,wherein said first system is one of a simulation and a physical system,and said second system is one of a simulation and a physical system. 11.A method to correlate operation of first and second systems comprising:generating a first set of metrics for a first set of data producedduring operation of said first system; generating a second set ofmetrics that correspond to each of said first set of metrics from asecond set of data produced during operation of said second system;computing a correlation score for said first and second systemsindicating a similarity of operation based on differences between saidfirst set of metrics and said second set of metrics; and presenting saidfirst set of metrics, said second set of metrics, and said correlationscore on a display to indicate said similarity of operation.
 12. Themethod of claim 11, further comprising normalizing said second set ofmetrics with respect to said first set of metrics such that said firstset of metrics forms a baseline.
 13. The method of claim 12, whereinpresenting comprises presenting said normalized second set of metrics onsaid display relative to said baseline.
 14. The method of claim 13,wherein presenting comprises presenting said normalized second set ofmetrics and said baseline as a radial plot comprising an axis for eachcorresponding metric, wherein each metric in said normalized second setof metrics is presented relative to a corresponding metric in saidbaseline.
 15. The method of claim 14, wherein presenting comprisespresenting said radial plot with a scale to indicate the relativedifferences between said second set of metrics and said first set ofmetrics.
 16. The method of claim 12, wherein computing comprisescomputing said correlation score based on differences between saidnormalized second set of metrics and said baseline.
 17. The method ofclaim 12, wherein computing comprises computing said correlation scorebased on weighted differences between said normalized second set ofmetrics and said baseline.
 18. The method of claim 11, whereinpresenting comprises presenting said first set of metrics and saidsecond set of metrics as a radial plot comprising an axis for eachcorresponding metric, wherein each metric in said second set of metricsis presented relative to a corresponding metric in said first set ofmetrics.
 19. The method of claim 11, further comprising weighting saidmetrics within said first and second sets of metrics.
 20. The method ofclaim 11, wherein said first system is one of a simulation and aphysical system, and said second system is one of a simulation and aphysical system.
 21. A computer-readable tangible medium encoded withcomputer program logic to correlate operation of first and secondsystems, said computer program logic comprising instructions, that whenexecuted by a processor, cause the processor to: generate a first set ofmetrics for a first set of data produced during operation of said firstsystem; generate a second set of metrics that correspond to each of saidfirst set of metrics from a second set of data produced during operationof said second system; compute a correlation score for said first andsecond systems indicating a similarity of operation based on differencesbetween said first set of metrics and said second set of metrics; andpresent said first set of metrics, said second set of metrics, and saidcorrelation score on a display to indicate said similarity of operation.22. The computer-readable tangible medium of claim 21, furthercomprising instructions to normalize said second set of metrics withrespect to said first set of metrics such that said first set of metricsforms a baseline.
 23. The computer-readable tangible medium of claim 22,wherein said instructions that present comprise instructions to presentsaid normalized second set of metrics on said display relative to saidbaseline.
 24. The computer-readable tangible medium of claim 23, whereinsaid instructions that present comprise instructions to present saidnormalized second set of metrics and said baseline as a radial plotcomprising an axis for each corresponding metric, wherein each metric insaid normalized second set of metrics is presented relative to acorresponding metric in said baseline.
 25. The computer-readabletangible medium of claim 24, wherein said instructions that presentcomprise instructions to present said radial plot with a scale toindicate the relative differences between said second set of metrics andsaid first set of metrics.
 26. The computer-readable tangible medium ofclaim 22, wherein said instructions that compute comprise instructionsto compute said correlation score based on differences between saidnormalized second set of metrics and said baseline.
 27. Thecomputer-readable tangible medium of claim 22, wherein said instructionsthat compute comprise instructions to compute said correlation scorebased on weighted differences between said normalized second set ofmetrics and said baseline.
 28. The computer-readable tangible medium ofclaim 21, wherein said instructions that present comprise instructionsto present said first set of metrics and said second set of metrics as aradial plot comprising an axis for each corresponding metric, whereineach metric in said second set of metrics is presented relative to acorresponding metric in said first set of metrics.
 29. Thecomputer-readable tangible medium of claim 21, further comprisinginstructions to weight said metrics within said first and second sets ofmetrics.